---
id: tracing
title: Distributed Tracing
---

Configuring Distributed Tracing (DT) will enable you to obtain a visualization
of the call paths that take place in order to process a request made to Hydra.
It's yet another tool that you can use to aid you in profiling, debugging and
ultimately understanding your deployment of Hydra better. Hydra currently
supports the following tracing options:

- Tracing backend(s): Jaeger, Elastic APM, Datadog, Zipkin and Instana - _Note:
  adding support for other
  [opentracing compliant backends](https://opentracing.io/docs/supported-tracers)
  is planned. To aid in priority, please
  [create an issue](https://github.com/ory/hydra/issues) with your feature
  request._
- Following existing traces: If you have deployed Hydra behind a proxy that has
  initiated a trace, Hydra will attempt to join that trace by examining the
  request headers for tracing context.

### What a Hydra trace includes

In DT speak, a trace is comprised of one or more spans which are logical units
of work. Each Hydra span is encapsulated with the following state:

- A name
- A start time
- A finish time
- A set of zero or more tags

Hydra currently creates the following spans:

- Top level span (_named after the request path_) for the requested endpoint.
  Span tags: - http method - http status code - error IFF status code >= 400
- Child span will be created if bcrypt (_e.g. when the token endpoint is
  called_) is called. Span tags: - bcrypt work factor
- All SQL database interactions. Spans/tags will vary depending on the database
  driver used.

This is still evolving and subject to change as tracing support continues to
expand in Hydra. If you see something that is missing/wrong, please create an
issue.

### Alright, how can I set this up locally?

The
[provided docker-compose file](https://github.com/ory/hydra/blob/master/quickstart-tracing.yml)
in the project repository has tracing configuration which you can use to play
around with - just uncomment the desired tracing provider. We will use Jaeger as
an example.

Simply run

```shell script
$ docker-compose -f quickstart.yml \
    -f quickstart-tracing.yml \
    up --build
```

Grab a coffee or stretch while you wait for everything to come up. You will then
be able to navigate to the Jaeger UI which you have exposed on port `16686` at
http://localhost:16686/search. You can now start making requests to Hydra and
inspect traces!

As an example, here is a trace created by making a bad request to the
`POST /clients` endpoint:

![OpenTracing and OpenCensus exemplary trace in Jaeger UI](../images/sample_trace.png)

At a glance, you are able to see that:

- The request failed
- The request took ~80ms
- It resulted in a 409
- The hash comparison to validate the client's credentials took a whopping 70ms.
  Bcrypt is expensive!
- The various database operations performed

_Note: in order to see spans around database interactions, you must be using a
SQL backend (i.e. MySQL or Postgres)._

### Tracing configurations

The CLI will provide you with the list of Hydra tracing configurations and their
supported values. Simply run:

```
docker exec -it hydra_hydra_1 hydra serve --help
```

And read the section on `DEBUG CONTROLS`.
